-
-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Testbed #94
base: master
Are you sure you want to change the base?
Testbed #94
Conversation
Initial Commit of the Testbed: There are likely several bugs and issues at this stage, but I wanted to avoid further delays. I plan to test the Google Colab integration tomorrow and will continue refining the setup based on the results. |
Since the previous PR is now merged, I did a quick rebase here and it looks like that all worked no problem. |
Well, about that 80/20 rule—80% of the code in 20% of the time? In my case, it's more like 10/90. I had most of the current functionality nailed down in the previous commit, everything working as intended on Linux. However, the assumption that Colab wouldn't be a problem turned out to be very wrong. Colab requires Python 10, while PanelCleaner is using Python 11. Image visualization in Colab is unacceptably slow, and this commit uses a lot of them. In general, everything was very slow and/or very buggy. I've had to refactor most of the code before time, move the _testbed folder inside pcleaner, and even implement a web server with a reverse proxy just to make the Colab experience smooth enough. Idefics quantized to 4 bits can run in most consumer-grades cards at or above 6GB and Colab free tier. I think the result is more robust, useful, and efficient. The framework is now in a state where it's much easier to add new experiments and tests. This was work, and now the (hopefully) fun part: Future Work (No particular order or schedule):
|
Wow, got a lot of plans, impressive! All those technical issues getting collab to work sound like an absolute nightmare. You have my condolences. About generating reports, I've got some experience generating LaTeX documents to make pdfs, then transforming that into html. But if the aesthetics aren't super important, then I'd definitely recommend generating HTML, then using a python package to dump the html into a pdf file automatically. It may not look amazing, but takes no effort. Let me know if you run into issues that need my attention, or would like to share some thought. |
Thank you for your insights on the report generation! I'll definitely consider the HTML to PDF route for simplicity. Could you please review the two final user (for developers, really) notebooks, If you’re interested and have some time, the I've successfully integrated the Paligemma model—amazing little model, remarkable performance and accuracy, pity about the restrictive license. It's outperforming Idefics2 and has achieved 98-99% accuracy without fine-tuning. This will likely be the last OCR model I test for a while as I shift focus to other functionalities. I’ll dive into the reporting aspect soon, but first, there’s some boring bookkeeping, like run comparisons, I need to tackle. Looking forward to your thoughts and any improvements you might have! |
Wow, the ExperimentsVisor is super cool! The I tried the idefics too, but unfortunately got stuck here:
And with that being in an auto generated 2k lines file (wow) I didn't figure out a way to fix it. Crazy impressive though! Can't wait to see Idefics in action for myself. Side note: |
One release later, I got the latest patches on master over here. Hope it doesn't cause conflicts. All platforms except Flatpak have your awesome update...I don't have time to open that can of worms today. |
Oops, just like with the first commit, I tested the Colab setup but overlooked the local install, which seems to have some issues. I'm currently in the middle of a refactor, but I'll push a commit with improved documentation and some fixes later today or tomorrow.
Absolutely, that line was intended for a Colab local dev install. I forgot to wrap it with
Not sure what the issue is yet, but I'm planning to strengthen and modify both
I completely agree about not pinning requirements. The issue with PIL in Colab is peculiar, though. It seems related to If we decide to continue with this PR and the testbed playground, we might consider creating a CLI-only package/wheel for installing PanelCleaner in Colab with leaner requirements. In Colab, we really only need the CLI components. By the way, it appears entirely feasible to launch PySide6/QT from Jupyter and simultaneously have the GUI, the CLI, and everything running in a notebook environment sharing the kernel and all the debugging tools. I haven't tested this yet, though. Right now, I'm more into the CLI than the GUI part of PanelCleaner. Sorry for the inconvenience. As a bonus, in the next commit, I'll include improvements to both Tesseract and Idefics, along with the addition of the awesome PaliGemma. |
Fixed the issue. The visor wasn't finding the test images. It's simply a matter of catching the fact earlier and informing the user |
That sounds awesome, looking forward to it. I'm not sure there is much to slim down in terms of having a CLI only build, I think it's just PySide6. It'd save 530MB in the unpacked state from the total of 2.1GB without CUDA, or the total of 7.3GB with CUDA, not to mention the 31GB of idefics, but at least pip doesn't need to deal with those. All of the assets and code for the GUI are a mere 2.7MB. I played around with the idea and it's actually trivial to generate such a whl, much to my surprise. |
I've updated the README with detailed instructions for local installation from scratch and thoroughly tested it on both mac and Linux to ensure everything is in good shape. I've also streamlined the Colab integration, which now doesn't require a full PanelCleaner. This did require a few tweaks outside the testbed folder however, specifically to the tr() function in helpers and the manga-ocr import. It's just a temporary fix to smooth things over for Colab users. This got me thinking about the possibility of a CLI-only install of PanelCleaner for Colab, which could be a neat solution for users without GPUs. I've also added PaliGemma, which seems to be a friendlier and more powerful model than Idefics, especially for OCR tasks. There have been a lot of changes and refactors aimed at stabilizing the current build, even though it's still somewhat limited. These updates should really help smooth the way for next steps. Please try installing clean and experimenting with the testbed both locally and in Colab to further refine the process and catch any bugs that may already be lurking in the shadows. |
Oops, forgot to mention the ground truth file format has changed. Re-download the source folder or delete the .txt files and add the new JSONs. |
So, I suppose this would help then? Looking like it was a busy and successful week! |
Hey Voxel, Glad that you're even considering the Colab stuff, overcoming your little love of notebooks 😇. Yeah, an option allowing us to do: Your findings about the significant reduction in the wheel file size are really promising! I'm no expert on python packages or wheels. Regarding the inclusion of dependencies, you're right that we don't need to bundle all the dependencies directly with the wheel. For a CLI-only tool, I think that is standard practice, isn't it? For apps and exes probably is a good idea to bundle all the dependencies. For a command-line tool, like pcleaner-CLI would be, I'm not sure if you've had a chance to try out Colab yet, but I'd really appreciate your thoughts on its usability and overall experience. The current setup is a bit hackish, I know, but here are the timings:
Model load times vary. In about 1.5 minutes, you can start working with PanelCleaner/testbed using a more or less powerful GPU from anywhere you have a browser on a PC, phone, or tablet. Besides Colab, there are many other free and paid notebook-based services like Kaggle, Azure, IBM Watson, Google Cloud, Amazon SageMaker, Databricks, etc. IMHO, I think it would be beneficial to have Colab support. That's the whole point of the testbed subpackage—to test new ideas not only for performance but also for their implications for inclusion in PanelCleaner main. However, you have to consider the audience of PanelCleaner. From my perspective, PanelCleaner targets two types of users:
Both groups would greatly benefit from notebook support. It would also help with the discovery of PanelCleaner. Imagine a Colab badge in the README or on the project page that launches PanelCleaner, ready to clean manga and showcasing all its capabilities before trying the app. And, on a personal note, as my Linux setup lacks a window manager, a CLI-only package would be much appreciated. How to proceed? I would open a new PR and explore four levels of integration. Before we get into that, let's review the options. This explanation will be long, so please bear with me. Note that in this discussion, I am not considering model downloads, which are unavoidable for non-local installations and must be performed each time. The libraries pre-installed in Colab are specifically tailored for data scientists and DL research, which might differ from other notebook platforms. I've already streamlined the PanelCleaner CLI requirements to exclude those already present in Colab (see Current solution with git, 12-15 seconds: (Ugly and hackish, but fast) if FC.IN_COLAB:
pc_path = 'PanelCleaner'
tb_path = pc_path/'pcleaner/_testbed'
!git clone -b testbed https://github.com/civvic/PanelCleaner.git
assert tb_path.exists(), "PanelCleaner not found"
os.chdir(tb_path)
sys.path.append(f"{pc_path.resolve()}")
sys.path.append(f"{tb_path.resolve()}")
!pip install -q -r requirements-colab.txt Using !pip install git+https://github.com/civvic/PanelCleaner.git@testbed Collecting git+https://github.com/civvic/PanelCleaner.git@testbed
Cloning https://github.com/civvic/PanelCleaner.git (to revision testbed) to /tmp/pip-req-build-7qvv_1fq
Running command git clone --filter=blob:none --quiet https://github.com/civvic/PanelCleaner.git /tmp/pip-req-build-7qvv_1fq
Running command git checkout -b testbed --track origin/testbed
Switched to a new branch 'testbed'
Branch 'testbed' set up to track remote branch 'testbed' from 'origin'.
Resolved https://github.com/civvic/PanelCleaner.git to commit 59177340547b0f3699d4396b19e8c9e5b3b012ad
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (4.8.0.76)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (4.41.1)
Collecting manga-ocr (from pcleaner==2.7.0)
Downloading manga_ocr-0.1.11-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.9/62.9 kB 2.0 MB/s eta 0:00:00
Requirement already satisfied: Pillow==9.4.0 in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (9.4.0)
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (2.3.0+cu121)
...
Requirement already satisfied: tifffile in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (2024.5.22)
Collecting PySide6>=6.7.0 (from pcleaner==2.7.0)
Downloading PySide6-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (530 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 530.7/530.7 kB 30.3 MB/s eta 0:00:00
Requirement already satisfied: attrs in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (23.2.0)
...
Collecting simple-lama-inpainting (from pcleaner==2.7.0)
Downloading simple_lama_inpainting-0.1.2-py3-none-any.whl (9.6 kB)
...
Collecting shiboken6==6.7.1 (from PySide6>=6.7.0->pcleaner==2.7.0)
Downloading shiboken6-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (188 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.4/188.4 kB 18.4 MB/s eta 0:00:00
Collecting PySide6-Essentials==6.7.1 (from PySide6>=6.7.0->pcleaner==2.7.0)
Downloading PySide6_Essentials-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (87.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.6/87.6 MB 8.3 MB/s eta
0:00:00
...
Preparing metadata (setup.py) ... done
Collecting fugashi (from manga-ocr->pcleaner==2.7.0)
Downloading fugashi-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (600 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.9/600.9 kB 34.3 MB/s eta 0:00:00
Collecting jaconv (from manga-ocr->pcleaner==2.7.0)
Downloading jaconv-0.3.4.tar.gz (16 kB)
Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of manga-ocr to determine which version is compatible with other requirements. This could take a while.
Collecting manga-ocr (from pcleaner==2.7.0)
Downloading manga_ocr-0.1.10-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 6.1 MB/s eta 0:00:00
Requirement already satisfied: pyperclip in /usr/local/lib/python3.10/dist-packages (from manga-ocr->pcleaner==2.7.0) (1.8.2)
Collecting unidic-lite (from manga-ocr->pcleaner==2.7.0)
Downloading unidic-lite-1.0.8.tar.gz (47.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 MB 11.5 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->pcleaner==2.7.0) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->pcleaner==2.7.0) (4.11.0)
...
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->pcleaner==2.7.0)
Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->pcleaner==2.7.0)
Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 51.8 MB/s eta 0:00:00
Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers->pcleaner==2.7.0) (0.23.1)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers->pcleaner==2.7.0) (24.0)
...
Collecting fire (from manga-ocr->pcleaner==2.7.0)
Downloading fire-0.5.0.tar.gz (88 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.3/88.3 kB 10.6 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of simple-lama-inpainting to determine which version is compatible with other requirements. This could take a while.
Collecting simple-lama-inpainting (from pcleaner==2.7.0)
Downloading simple_lama_inpainting-0.1.1-py3-none-any.whl (9.6 kB)
Downloading simple_lama_inpainting-0.1.0-py3-none-any.whl (13 kB)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from fire->manga-ocr->pcleaner==2.7.0) (1.16.0)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire->manga-ocr->pcleaner==2.7.0) (2.4.0)
...
Building wheels for collected packages: pcleaner, fire, jaconv, unidic-lite
Building wheel for pcleaner (pyproject.toml) ... done
Created wheel for pcleaner: filename=pcleaner-2.7.0-py3-none-any.whl size=1349831 sha256=272b31f9f54c164950f83514612d8b26fdf5870a45530743940183e4997393c8
Stored in directory: /tmp/pip-ephem-wheel-cache-f4rz_d29/wheels/84/e3/8c/b198fffc383f392c64a5c379c7c252a362221e00be0d0160e2
...
Successfully built pcleaner fire jaconv unidic-lite
Installing collected packages: unidic-lite, pyclipper, jaconv, dictdiffer, xdg, shiboken6, python-magic, pytesseract, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, ...
... Of those 2 minutes, more than half is for Probably, but have to check, it would be faster to install the two models, pip install manga-ocr
pip install simple-lama-inpainting
pip install pcleaner-CLI So, have an option with only the reqs for Colab, Apart from that decision to take beforehand, the next phases in a possible new PR would be:
In my view, PanelCleaner is a specialized tool for manga/comics processing that leverages deep learning. You surely have a different perspective, and as the creator, your vision is what matters. Nonetheless, PanelCleaner has the potential to evolve into a multi-platform tool that caters to diverse needs in the specialized domain of comics, offering various interfaces like command line, GUI, and notebooks. This evolution should be gradual and pressure-free, this is a hobby. Before all that, it's important to acknowledge the current state of PanelCleaner. It's a robust, self-contained package that's stable and performs well, designed with specific functionalities and audiences in mind. Expanding its scope is certainly exciting and fun work, but it also introduces potential for more bugs, more issues, and increased support demands. We need to consider whether we are willing or able to dedicate more time to new stuff. Maintaining open-source packages is rewarding yet demanding. Too ambitious? Am I getting ahead of myself? |
BTW, I plan to include manga-ocr in the testbed soon for comparison purposes. I'll check for any dependency collisions at that time. Also, could you spare a few minutes to help create the ground truth for one or two of the Japanese pages in the testbed set? I'm not quite sure how to start with that. Also, I'm no expert in manga as your are, maybe we should include more representative pages than the ones I selected in a hurry. |
Whoa, that's a lotta info. Yes, then I'll be pushing my changes to have a It is important to understand what a whl is. This is a zip file containing the source code and some metadata that pip understands so it can install the package for you. The bigger win is no longer needing PySide6, which saves 530MB. The remaining dependencies stil need 1.5GB, or roughly 7GB if CUDA. If you wish to limit dependency overlap, you do not need to make a special package for that. Instead, use Thus, pcleaner-cli is necessary because we need to excise the gui code in its entirety to safely run without pyside, but a pcleaner-cli-colab that simply has less dependencies is not sensible. --no-deps and a custom requirements list take care of that much better. As for the following points.
Yeah, Panel Cleaner is a tool for primarily cleaning and uses deep learning at the core. It's already multi platform and offers a cli and gui, not sure what sector exactly you wish to expand to, other than notebooks? I won't do web development so that's kinda where it ends for me at least. |
I'm honestly no expert either and don't know any Japanese, but I know a guy who is and I know another guy who can translate. I'll arrange something in the coming days. Very impressive what you're making. |
That settles the issue, then. I concur; it's a sensible solution. I should have looked a bit deeper into pip and Python packages, but until now, I hadn't had the need.
Absolutely, I continue to believe the MVC paradigm is as relevant today as it was when SmallTalk popularized it. I meant adding additional code in the notebooks. These levels refer to progressively complicating the notebooks. If someday this PR goes mainstream, the notebooks can reuse code from the testbed. If not, the notebooks will remain isolated, utilizing pcleaner's main functionality and adding new code as deemed necessary by the notebook developer.
I was suggesting using a different localization system, not having two of them. But I'm no expert; it's been many years since I built my last cross-platform or native app (thank the gods, what a can of worms!). Localization or GUIs in general is not a field I'm interested in. If you agree to keep English as the only language of the CLI (when installed with pcleaner-cli), that works perfectly for me.
Your answer has provided a ready solution. In the Idefics notebook, I only install Idefics, not manga-ocr or Tesseract. In the notebook that needs manga-ocr, I will install pcleaner-cli without the manga-ocr dependency. Then, I will install manga-ocr separately without dependencies, and last install the rest of pcleaner + manga-ocr dependencies separately. Essentially, it would be the same solution I'm using now, but with the neat pcleaner-cli instead of the git hack.
No worries, I'm not particularly interested in Gradio; there's little fun there for me. I have to learn about it because nowadays, almost all work in AI is showcased with Gradio. Almost every new paper, research, or new DL-based product has an associated HuggingFace Space with Gradio, even the big players like Google, Meta, OpenAI, and soon Apple. As with notebooks, supporting it should not involve changing anything in pcleaner main—apart from the minor points we are now contemplating. If you want to extend the audience of PanelCleaner, having both a CLI and a GUI is fantastic. Adding notebooks and a HuggingFace Space could be also a nice touch, providing new users additional avenues to discover, try out, and evaluate PanelCleaner. I won't be the one to do it, though; I'll leave that to others.
And finally, the main point of my comment (I apologize for digressing earlier):
If you're not interested, that's fine, I'd prefer to spend my time on more enjoyable DL/AI stuff along the lines of this PR. |
I have now published the pcleaner-cli (it's on pypi, so You can continue working on building out the notebook side, if you prefer. No need to split off the foundation just for the sake of it. As for |
I just realized that python 3.10 doesn't support StrEnum, I figured this was for colab running python 3.9 or so. Turns out there is already a compatibility package for this exact purpose, so the ugly hacks to remove StrEnum aren't necessary. Please revert "Downgrade to Python 10 as Colab not yet support" 11 a2d15f4 |
…format; start refactorin Experiment_run
This reverts commit a2d15f4.
I took care of it, sorry for wasting your time with civvic#1 that was stupid of me. Let me know if the force push broke anything. A rebase is generally better, but not always when collaborating. Be sure to backup your code before fetching. |
I'm now officially befuddled and bewildered 🤪 'cause I don't exactly know what just happened, but all is good in the world and seems to roll ok 👏 I'm adding pip and git to the ever-growing list of things I got to look into a lot more |
Support Python3.10 again didn't patch structures.py, will fail for those running 3.10 |
Good to hear that it works. The rebase is a maneuver to pretend you split off your branch from a different point than you actually did, which adds or removes commits from the base branch into your branch. Whereas the stupid thing I blindly followed github's directions into doing was to just contribute to your fork, instead of contributing to this branch right here in my own project that you opened. Git is very powerful, it's worth knowing how to do a git rebase properly. Looks like I need to do it all again then. No clue how that slipped past a global search, but I'll fix it. I don't have python 3.10 anymore, maybe I should get it back for testing. |
I did the lazy thing and used a merge commit this time. It should be ok. All fixed now. |
Seeing as any more commits to master will be a lot of effort to rebase through with the conflicts I've caused today, it may be wise to take the time to put the notebook foundations in place before building even more on top, just to keep git manageable. That isn't to say it needs to be done right now, I'll leave that up to you. Before that though, I'd like to know if it's possible to move the _testbed folder out of the pcleaner directory, since it isn't actually part of the package that gets distributed. What was the original issue that necessitated this placement? I suspect it may have been fixed with the introduction of the pcleaner-cli package, but that's just a hunch. Also, I think it'd still be better to rename it to testbed, without hiding it with |
Introduction of a Possible PanelCleaner Testbed
Overview
This Pull Request establishes the _testbed directory within the PanelCleaner project. It is designed as a separate space for developers and maintainers to experiment with and test new ideas, technologies, and methodologies that could influence future directions of PanelCleaner.
Key Features
ocr_metric.ipynb
andexperiments.ipynb
are included to define metrics and document the evaluation processes.Impact
The creation of the _testbed directory can be a move to enhance the development process of PanelCleaner and allow collaborations by providing a structured environment for innovation and testing. This will aid in quickly iterating over new ideas and integrating successful ones into the main project.
Future Directions
The
testbed
should evolve based on the experiments conducted and the feedback from developers. It will continuously adapt to include more sophisticated tools and frameworks that align with the evolving field of deep learning.